Method and system for medical imaging evaluation

ABSTRACT

This disclosure generally pertains to methods and systems for processing electronic data obtained from imaging or other diagnostic and evaluative medical procedures. Certain embodiments relate to methods for the development of deep learning algorithms that perform machine recognition of specific features and conditions in imaging and other medical data. Another embodiment provides systems configured to detect and localize medical abnormalities on medical imaging scans by a deep learning algorithm.

RELATED APPLICATIONS

This application claims priority benefit of Indian Patent ApplicationNo. 202021001232, filed Jan. 10, 2020, which are incorporated entirelyby reference herein for all purposes.

TECHNICAL FIELD

This disclosure generally pertains to methods and systems for processingelectronic data obtained from imaging or other diagnostic and evaluativemedical procedures. Some embodiments relate to methods for thedevelopment of deep learning algorithms that perform machine recognitionof specific features and conditions in imaging and other medical data.

BACKGROUND ART

Medical imaging techniques, such as computed topography (CT) and X-rayimaging, are widely used in diagnosis, clinical studies and treatmentplanning. There is an emerging need for automated approaches to improvethe efficiency, accuracy and cost effectiveness of the medical imagingevaluation.

Chest X-rays are among the most common radiology diagnostic tests, withmillions of scans performed globally every year. While the test isfrequently performed, reading chest X-rays is among the more complexradiology tasks, and is known to be highly subjective, with inter-readeragreement varying from a kappa value of 0.2 to 0.77, depending on thelevel of experience of the reader, the abnormality being detected andthe clinical setting.

Due to their affordability, chest X-rays are used all over the world,including in areas with few or no radiologists. In many parts of theworld, the availability of digital chest X-ray machines is growing morerapidly than the availability of clinicians who are trained highlyenough to perform this complex task. If automated detection can beapplied in low-resource settings as a disease screening tool, thebenefits to population health outcomes globally could be significant.One example of such use of chest X-rays is in tuberculosis screening,where chest X-rays, in the hands of expert readers are more sensitivethan clinical symptoms for the early detection of tuberculosis.

Over the last few years, there has been increasing interest in the useof deep learning algorithms to assist with abnormality detection onmedical images. This is a natural consequence of the rapidly growingability of machines to interpret natural images and detect objects inthem. On chest X-rays in particular, there have been a series of studiesdescribing the use of deep learning algorithms to detect variousabnormalities (Shin, et al., Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 2497-2506, 2016). Most ofthese have been limited by the lack of availability of largehigh-quality datasets, with the largest published work describing analgorithm that has been trained with 112,120 X-rays, a relatively smallnumber considering that the majority of chest X-rays are normal, andabnormal X-rays are less common, with specific abnormalities being rarerstill.

SUMMARY OF THE INVENTION

The present disclosure describes the development and clinical validationof fully automated deep learning systems that are trained to detect andlocalize abnormalities from medical imaging scans.

Certain embodiment provides the training and clinical validation of adeep learning system to detect and localize chest X-ray abnormalities.The proprietary system has been trained on 2.5 million chest X-rays andtested it against the majority vote of a panel of 6 radiologists on anindependent dataset containing 2000 studies. Abnormalities on chestX-rays range from very small lesions to diffuse abnormalities that coverlarge parts of the lung field.

In particular, an embodiment provides a method for developing a deeplearning system to detect and localize medical abnormalities on chestX-ray imaging scans, comprising:

selecting chest X-ray imaging scans and corresponding radiology reportsand extracting the medical abnormalities via Natural language processing(NLP) algorithms to generate extracted findings, wherein the extractedfindings are used as labels for training a deep learning algorithm;

segmenting, via an anatomy segmenter, the selected chest X-ray imagingscans to generate segmentation masks corresponding to chest cavity,lungs, diaphragm, mediastinum and ribs;

outputting, via a region of interest (ROI) generator, a plurality ofROIs that are relevant for detecting a particular abnormality, whereinthe ROI generator uses the chest X-ray imaging scan at full resolutionand the corresponding anatomy segmentation masks; and

detecting the abnormalities, via abnormality detector, and outputting alow-resolution probability map per each ROI, a confidence score per eachROI, and a confidence score for an entire chest X-ray scan by combiningthe confidence scores per each ROI, wherein the abnormality detector isa hybrid classification plus segmentation network.

According to one embodiment, the extracted findings comprise locations,severity, size, shape and texture. The labels comprise scan-levellabels, ROI level labels and pixel level labels.

According to another embodiment, the NLP algorithms are rule-based. Thedeep learning algorithm comprises convolutional neural networks (CNNs).

In at least one embodiment, the anatomy segmenter uses a U-Net basedneural network. The ROI generator is rule-based. The abnormalitydetector is based on ResNeXT-50 with Squeeze-excitation modules(SE-ResNeXT-50).

In at least one embodiment, the system uses a weighted sum of crossentropy loss at three levels comprising ROI level predictions, pixellevel probability maps, and final pooled chest X-ray scan levelprediction. The confidence scores per each ROI are combined using apooling operator that is a convex approximation of LogSumExp (LSE)function.

According to an embodiment, the said medical imaging scans include butnot limited to CT, X-ray, magnetic resonance imaging (MRI), andultrasound procedures. For chest X-ray scans, the said medicalabnormalities include but not limited to blunted CP angle,calcification, cardiomegaly, cavity, consolidation, fibrosis, hilarenlargement, opacity and pleural effusion.

Another embodiment provides a system for automating detection andlocalization of medical abnormalities on chest X-ray imaging scans usinga deep learning algorithm carried out by a computer, wherein the deeplearning algorithm is developed by the steps of:

selecting chest X-ray imaging scans and corresponding radiology reportsand extracting the medical abnormalities via Natural language processing(NLP) algorithms to generate extracted findings, wherein the extractedfindings are used as labels for training a deep learning algorithm;

segmenting, via an anatomy segmenter, the selected chest X-ray imagingscans to generate segmentation masks corresponding to chest cavity,lungs, diaphragm, mediastinum and ribs;

outputting, via a region of interest (ROI) generator, a plurality ofROIs that are relevant for detecting a particular abnormality, whereinthe ROI generator uses the chest X-ray imaging scan at full resolutionand the corresponding anatomy segmentation masks; and

detecting the abnormalities, via abnormality detector, and outputting alow-resolution probability map per each ROI, a confidence score per eachROI, and a confidence score for an entire chest X-ray scan by combiningthe confidence scores per each ROI, wherein the abnormality detector isa hybrid classification plus segmentation network.

Adventageous Effects of the Invention

The present invention provides a novel solution in processinglarge-scale medical imaging data in a computationally-efficient mannerand enables a scalable and automated system and method to create labelsfor training a deep learning system that can output abnormalitylocations in medical imaging scans.

Previous published work on deep learning for chest X-ray abnormalitydetection has not made a distinction between the diagnosis of “diseases”and the detection of “abnormal findings”. The present invention is tofocus on the detection of abnormal findings, or abnormalities on theX-ray that can be detected visually by an expert without any priorknowledge of clinical history. This allows the system to be appliedacross geographical settings with different disease prevalence patternsand across different disease manifestations.

BRIEF DESCRIPTION OF DRAWINGS

The invention will be described in more detail below on the basis of adrawing, which illustrates exemplary embodiments. In the drawing, ineach case schematically:

FIG. 1 Exemplary work flow of the deep learning system.

FIG. 2. Proposed workflow of abnormality detection.

FIG. 3. Proposed workflow for TB detection using qXR for pilot atdistrict TB centers.

DETAILED DESCRIPTION

It should be understood that this invention is not limited to theparticular methodology, protocols, and systems, etc., described hereinand as such may vary. The terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to limit thescope of the present invention, which is defined solely by the claims.

As used in the specification and appended claims, unless specified tothe contrary, the following terms have the meaning indicated below.

“Architecture” refers to a set of rules and methods that describe thefunctionality, organization, and implementation of computer systems.

“Convolutional neural network (CNN)” refers to a class of deep,feed-forward artificial neural networks, most commonly applied toanalyzing visual imagery. CNNs use a variation of multilayer perceptronsdesigned to require minimal preprocessing. A CNN consists of an inputand an output layer, as well as multiple hidden layers. The hiddenlayers of a CNN typically consist of convolutional layers, poolinglayers, fully connected layers and normalization layers. Convolutionallayers apply a convolution operation to the input, passing the result tothe next layer. Local or global pooling layers combine the outputs ofneuron clusters at one layer into a single neuron in the next layer.Fully connected layers connect every neuron in one layer to every neuronin another layer. CNNs use relatively little pre-processing compared toother image classification algorithms. This means that the networklearns the filters that in traditional algorithms were hand-engineered.This independence from prior knowledge and human effort in featuredesign is a major advantage.

“Heuristics” refers to a technique designed for solving a problem morequickly when classic methods are too slow, or for finding an approximatesolution when classic methods fail to find any exact solution. This isachieved by trading optimality, completeness, accuracy, or precision forspeed. In a way, it can be considered a shortcut. A heuristic function,also called simply a heuristic, is a function that ranks alternatives insearch algorithms at each branching step based on available informationto decide which branch to follow. The objective of a heuristic is toproduce a solution in a reasonable time frame that is good enough forsolving the problem at hand. This solution may not be the best of allthe solutions to this problem, or it may simply approximate the exactsolution.

“Natural Language Processing (NLP)” refers to a way for computers toanalyze, understand, and derive meaning from human language in a smartand useful way. By utilizing NLP, developers can organize and structureknowledge to perform tasks such as automatic summarization, translationnamed entity recognition, relationship extraction, sentiment analysis,speech recognition, and topic segmentation.

“LogSumExp (LSE) function” refers to a smooth maximum: a smoothapproximation to the maximum function, mainly used by machine learningalgorithms. See Nielsen, Frank, et. al., Guaranteed bounds on theKullback-Leibler divergence of univariate mixtures using piecewiselog-sum-exp inequalities, Entropy. 2016.

“Cross entropy loss” measures the performance of a classification modelwhose output is a probability value between 0 and 1. Cross-entropy lossincreases as the predicted probability diverges from the actual label.So predicting a probability of 0.012 when the actual observation labelis 1 would be bad and result in a high loss value. A perfect model wouldhave a log loss of 0.

The present disclosure illustrates various techniques and configurationsthat enable the integration and use of machine learning analysis in adata-driven image evaluation workflow. For example, machine learninganalysis (such as trained models of image detection of certain medicalconditions) may be performed upon medical imaging procedure dataproduced as part of a medical imaging study. The medical imagingprocedure data may include image data captured by an imaging modality,and order data (such as data indicating a request for a radiologicalimage read), each produced to facilitate a medical imaging evaluation(such as a radiology read to be performed by a radiologist or adiagnostic evaluation by another qualified medical professional).

For example, the machine learning analysis may receive and processimages from medical imaging procedure data, to identify trainedstructures, conditions, and conditions within images of a particularstudy. The machine learning analysis may result in the automateddetection, indication, or confirmation of certain medical conditionswithin the images, such as the detection of urgent or life-criticalmedical conditions, clinically serious abnormalities, and other keyfindings. Based on the result of the machine learning analysis, themedical evaluation for the images and the associated imaging proceduremay be prioritized, or otherwise changed or modified. Further, thedetection of the medical conditions may be used to assist the assignmentof the medical imaging data to particular evaluators, the evaluationprocess for the medical imaging data, or implement other actions priorto, or concurrent with, the medical imaging evaluation (or thegeneration of a data item such as a report from such medical imagingevaluation).

As further discussed herein, the machine learning analysis may beprovided on behalf of any number of machine learning algorithms andtrained models, including but not limited to deep learning models (alsoknown as deep machine learning, or hierarchical models) that have beentrained to perform image recognition tasks, particularly for certaintypes of medical conditions upon medical images of human anatomy andanatomical representations. As used herein, the term “machine learning”is used to refer to the various classes of artificial intelligencealgorithms and algorithm-driven approaches that are capable ofperforming machine-driven (e.g., computer-aided) identification oftrained structures, with the term “deep learning” referring to amultiple-level operation of such machine learning algorithms usingmultiple levels of representation and abstraction. However, it will beapparent that the role of the machine learning algorithms that areapplied, used, and configured in the presently described medical imagingevaluation may be supplemented or substituted by any number of otheralgorithm-based approaches, including variations of artificial neuralnetworks, learning-capable algorithms, trainable object classifications,and other artificial intelligence processing techniques.

In some of the following examples, reference is made to radiologymedical imaging procedures (e.g., computed tomography (CT), magneticresonance imaging (MRI), Ultrasound, and X-ray procedures, etc.) anddiagnostic evaluation of the images produced from such imagingprocedures that would be performed with an image evaluation (e.g.,radiology read) by a licensed and credentialed radiologist. It will beunderstood that the applicability of the presently described techniquesand systems will extend to a wide variety of imaging data (and otherdata representations) produced by various medical procedures andspecialties, including those not involving traditional radiology imagingmodalities. Such specialties include, but are not limited, to pathology,medical photography, medical data measurements such aselectroencephalography (EEG) and electrocardiography (EKG) procedures,cardiology data, neuroscience data, preclinical imaging, and other datacollection procedures occurring in connection with telemedicine,telepathology, remote diagnostics, and other applications of medicalprocedures and medical science. Accordingly, the performance of the datarecognition and workflow modification techniques described herein mayapply to a variety of medical image data types, settings, and use cases,including captured static images and multi-image (e.g. video)representations.

The following description and the drawings sufficiently illustratespecific embodiments to enable those skilled in the art to practicethem. Other embodiments may incorporate structural, logical, electrical,process, and other changes. Portions and features of some embodimentsmay be included in, or substituted for, those of other embodiments.

FIG. 1 illustrates the exemplary work flow of the deep learning system.More than 2.5 millions Chest X-Ray scans and their correspondingradiology reports were used to train convolutional neural networks(CNNs) to identify the abnormalities. The X-ray scans together with theradiology reports are collectively called imaging procedure data.Natural language processing (NLP) algorithms were developed to parseunstructured radiology reports and extract information about thepresence of abnormalities in the chest X-ray scans. These extractedfindings were used as labels when training CNNs. The extracted findingscomprise locations, severity, size, shape and texture. The labelscomprise scan-level labels, ROI level labels and pixel level labels.

The NLP algorithm was constructed using a thoracic imaging glossary,curated by a panel of radiologists and tailored to be consistent withthe predefined abnormality definitions. This algorithm is rule-based asopposed to machine-learning based NLP. See John Zech, et. al., NaturalLanguage-based Machine Learning Models for the Annotation of ClinicalRadiology Reports, Radiology (2018). Rule-based systems performed betterthan learned methods, probably because of the vast amount of domainspecific knowledge that had to be imparted which would require largeamounts of annotated data. The proprietary NLP algorithm is essentiallya large set of rules.

Since data was collected from multiple sources, the reporting standardswere not consistent. The same finding can be noted in several differentformats. For example, the finding Blunted Costophrenic angle can bereported in either of the following ways: “CP angle is obliterated”;“Hazy costophrenic angles”; or “Obscured CP angle”. The system collectedall the wordings that can be used to report findings and created a rulefor each finding. As an illustration, the following rule can be used tocapture the above three variations of blunted CP angle:

((angle & (blunt|obscur|oblitera|haz|opaci))

The rule would be positive if there are words angle and blunted or theirsynonyms in a sentence. In addition to such rules, there can behierarchical structure in findings. For example, opacity is consideredpositive if any of edema, consolidation, groundglass, etc. are positive.The system therefore created a ontology of findings and rules to dealwith this hierarchy. As an illustration, the following rule capturesthis hierarchy for opacity.

[opacity]

rule=((opacit & !(/ & collapse))|infiltrate|hyperdensit)

hierarchy=(edema|groundglass|consolidation| . . . )

In addition to these rules that pick mentions of abnormal findings fromreports, to obtain the final labels, the system performs negationdetection, uncertainty detection and a set of standard NLP techniques toaccount for formatting and grammatical issues. The system also extractsqualifiers like left, right, upper zone, etc. that serve as additionallabels for a given image. The final algorithm is validated by expertradiologists. A group of 5 experts were given the abnormalitydefinitions. A dataset of 1930 reports along with their chest X-rayswere presented to this group and the findings extracted by the NLPalgorithm were compared against the findings extracted by the experts.Results from this validation are presented in the main text.

Table 1 lists definitions that were used when extracting radiologicalfindings from the reports.

TABLE 1 Abnormality definitions Definition for tag extraction fromFinding radiology reports Definition for radiologist review Normal ‘Noabnormality Normal X-ray detected’ or ‘Normal’ Blunted CP Blunted CPangle CP Angle blunted or obscured. angle Could be due to pleuraleffusion or pleural thickening Calcification Calcification Allcalcifications on X-ray including but not limited to aortic archcalcification, rib calcifications, calcified pulmonary densities andmicrocalcifications Cardiomegaly Cardiomegaly Cardiothoracic ratio > 0.5Cavity Pulmonary cavity Pulmonary cavity Consolidation Consolidation,Pulmonary consolidation pneumonia or air- bronchogram Fibrosis FibrosisAny evidence of lung fibrosis, including interstitial fibrosis,fibrocavitary lesion Hilar Hilar enlargement, Enlarged or prominenthilum or prominence prominent hilum or including hilar lymphadenopathyhilar lymphadenopathy Opacity Any lung field opacity Any lung fieldopacity or or opacities, shadow or multiple opacities including butdensity including but not limited to infiltrate, not limited toinfiltrate, consolidation, mass, nodule, consolidation, mass, pulmonarycalcification, and nodule, pulmonary fibrosis. Pleural abnormalities notcalcification, and included under this tag fibrosis Pleural PleuralEffusion Pleural Effusion Effusion

These findings are referred as tags. Tag extraction accuracy wasmeasured versus a set of reports where abnormalities were manuallyextracted. Tag extraction accuracy was reported in Table 2.

TABLE 2 Tagextraction accuracy. Sensitivity Specificity Finding#Positives (95% CI) (95% CI) Normal (No 105 0.9429 1.0000 abnormalitydetected (0.8798-0.9787) (0.9959-1.0000) Blunted CP angle 146 0.97950.9824 (0.9411-0.9957) (0.9712-0.9901) Calcification 116 1.0000 0.9660(0.9687-1.0000) (0.9519-0.9770) Cardiomegaly 125 0.9920 0.9760(0.9562-0.9998) (0.9635-0.9851) Cavity  30 1.0000 0.9856 (0.8843-1.0000)(0.9759-0.9921) Consolidation 161 0.9876 0.9761 (0.9558-0.9985)(0.9634-0.9854) Fibrosis 124 0.9839 0.9931 (0.9430-0.9980)(0.9851-0.9975) Hilar Enlargement 289 0.9689 0.9732 (0.9417-0.9857)(0.9585-0.9838) Opacity 612 0.9608 0.9251 (0.9422-0.9747)(0.8942-0.9492) Pleural Effusion 246 0.9309 0.9602 (0.8917-0.9592)(0.9436-0.9730) Total (all findings) 1954  0.9672 0.9771 (0.9584-0.9747)(0.9736-0.9803)

The next step is to train the deep learning system for abnormalitydetection. The optimal deep learning algorithm architecture differsbased on the abnormality being detected, hence separate detectionalgorithms for each abnormality are trained. A particular abnormalitydetection pipeline comprises: anatomy segmenter, Region of Interest(ROI) generator, and abnormality detector. See. FIG. 2. Given a chestX-ray, the anatomy segmenter outputs segmentation maps to extract lungs,diaphragm, mediastinum and ribs. The ROI generator uses the chest X-rayat full resolution and the corresponding anatomy segmentation masks tooutput a set of ROIs that are most relevant for detecting a particularabnormality. The abnormality detector is a hybrid classification plussegmentation network that uses the above set of ROIs. The abnormalitydetector produces outputs including a low-resolution probability map perROI, a list of confidence scores, one per ROI, and a confidence scorefor the entire chest X-ray exam by combining the list of per ROIconfidence scores.

To build the anatomy segmenter, a set of 5000 chest X-rays wereannotated at the pixel level with each of the above anatomical labels. AU-Net based neural network was trained to output anatomical segmentationmasks corresponding to the lungs, diaphragm, mediastinum and ribs. SeeOlaf Ronneberger, et. al., U-net: Convolutional networks for biomedicalimage segmentation. Lecture Notes in Computer Science, pages 234-241(2015). These segmentation networks operate on 256×256 resized versionsof the chest X-ray.

The ROI generator module is completely rule-based and is developed inclose collaboration with radiologists. The output of this module variesdepending on the abnormality detected. As an example, for pleuraleffusion, this module generates two ROIs that cover the lower halves ofthe lung along with the costophrenic angles and a part of the diaphragm.For lung parenchymal abnormalities, the ROI generator outputs ROIs thattogether cover the lungs from apex to the diaphragm. For cardiomegaly,the ROI generated is a single image of the entire chest X-ray resized to320×320 pixels. Each of these ROIs are then resized to a standard sizeof 320×320 pixels before being fed into the abnormality detector. Beforethe ROIs are generated, the original chest X-ray is resized to 2000×2000pixels.

The use of ROI generator to filter out regions in the x-ray that are notessential for a particular abnormality detection model enables the useof significantly higher resolution for the final classificationnetworks. For example, the pleural effusion detection model sees thecostophrenic angles and other relevant anatomy at a resolution that issignificantly higher than a model that uses the entire chest X-ray asinput.

The abnormality detector is based on ResNeXT-50 with Squeeze-excitationmodules (SE-ResNeXT-50). See Saining Xie, et. al., Aggregated residualtransformations for deep neural networks, arXiv preprint arXiv:1611.05431, 2016. These modules are modified to operate on images thatare significantly larger and use multiple types of labels (hybrid)during training. This module operates on the set of ROIs (320×320pixels) generated and outputs a probability map (40×40 pixels) and aconfidence score per ROI. These per ROI confidence scores are combinedusing a pooling operator that is a convex approximation of the maxfunction, for example, the LogSumExp (LSE) function.

The proprietary deep learning system uses a weighted sum of crossentropy loss at 3 levels comprising on the ROI level predictions; on thepixel level probability maps; and on the final pooled chest X-ray examlevel prediction. The weights for each of the losses werehyper-parameters that were tuned while training.

Chest X-ray level labels for a particular abnormality were available forthe entire dataset via the NLP labeler. Labels at other levels were notavailable for the entire dataset.

The ground truths for ROI level predictions are a combination of expertannotations and NLP generated labels where possible. For example, in acase of a chest X-ray with pleural effusion, the reporting radiologistgenerally mentions either left, right or bilateral while reporting theabnormality. Since, these qualifications are extracted by NLP algorithmswhile parsing the reports, it enabled to label ROIs in a scalablefashion. The percentage of abnormal x-rays which NLP algorithms wereable to label the ROIs ranged from 25-60% across abnormalities. The ROIsfrom x-rays that were reported as normal by the reporting radiologistwere labelled automatically as not containing that particularabnormality.

The ground truths for pixel-level probability maps are free-handannotations done by experts using a custom annotation portal. This isdone for approximately 20% of all the abnormal scans (˜6% of the entiredataset).

ROI level loss and pixel level loss were set to 0 wherever labels wereunavailable. All image ROIs were normalized to have pixel values between0 and 1 before being presented to the network. Our data augmentationstrategy consisted of random crops, brightness, contrast, gammaaugmentations and a set of abnormality specific augmentations. SeeKaiming He, et. al., Deep residual learning for image recognition, InProceedings of the IEEE conference on computer vision and patternrecognition, 2016. The aim of data augmentation is to train networksthat are unaffected by variability in X-ray machine manufacturer, model,voltage, exposure and other parameters that vary from center to center.Training was done on NVIDIA GPUs using the pytorch framework. Multiplemodels were trained by varying the model initialization conditions andthe distribution of the training dataset and a simple majority scheme isused to combine these models.

When the chest X-ray level prediction is positive for a particularabnormality, all the ROIs that are above a certain threshold are drawnas bounding boxes on the original X-ray for the purpose of highlightingthe region responsible for the positive X-ray level prediction. Inaddition to this, for subtle and small abnormalities like nodule, the40×40 probability map is thresholded and resized to 320×320 and overlaidon the positive ROI. Visualization is not generated for cardiomegaly asit is a single ROI.

The proprietary system uses an operating point where sensitivity isclosest to 0.95. with specificity >0. Otherwise, the system use anoperating point where sensitivity is just above 0.90 if available, elsethe closest to 0.90.

The proprietary deep learning system for automating detection of medicalabnormalities on chest X-ray scans particularly address the problem onscalability, sparsity, generalizability.

Annotation is generally limited by the time and expert effort requiredto label. This system uses a hybrid model for annotation: for a smallpart of the dataset, annotation is made at pixel, slice and boxeslevels; for other images, annotation is made at the scan level whereexpect annotations are unavailable. Based on the types of labelsavailable, the system uses multi-pronged supervised learning algorithmsto define a feature type as an abnormality.

Convolutional layers in a convolutional neural network (CNN) summarizethe presence of features in an input image. Pooling layers provide anapproach to down sampling feature maps by summarizing the presence offeatures in patches of the feature map. Two common pooling methods areaverage pooling and max pooling that summarize the average presence of afeature and the most activated presence of a feature respectively. Tosolve the sparsity problems, the system use a softmax pooling methodthat is between average and max pooling.

To address generalizability, the system removes irrelevant parts of theimage and increase variance in the dataset artificially by adjustingbrightness, contrast, gamma correction, random crop, or scale.

EXAMPLES Example 1. Deep Learning Solution for Tuberculosis Detection

The system is designed to screen and prioritize abnormal chest X-rays.FIG. 3 illustrates the exemplary workflow of using the deep learningsystem for Tuberculosis detection. The algorithm automaticallyidentifies 15 most common chest X-ray abnormalities. A subset of theseabnormalities that suggest typical or atypical pulmonary Tuberculosisare combined to generate a “Tuberculosis screening” algorithm within theproduct. The tuberculosis screening algorithm is intended to replicate aradiologist or physician's screen of chest X-rays for abnormalitiessuggestive of Tuberculosis, before microbiological confirmation. Thesystem integrates with Vendor Neutral Integration Process and works withX-rays generated from any X-ray system (CR or DR). The system screensfor Tuberculosis and also identifies 15 other abnormalities, so thatpatients can be informed about non-TB conditions they might be sufferingfrom.

The system seamlessly integrates with Vendor Neutral Archives (VNA) andPACS without interfering with the existing Radiology workflow. It isdeployed in HIPAA compliant cloud servers to deliver a table of resultscontaining:

-   -   a. “Normal” or “abnormal” for each chest X-ray    -   b. List of all abnormalities detected by the algorithm with its        corresponding probability scores.    -   c. “Tuberculosis screen advised” or “tuberculosis screen        negative” for each X-ray, with probability scores.

Using standard imaging protocols, the system automatically uploads theX-rays of interest (anonymized) from the client PACS/VNA and respondsback with the results overlay and corresponding reports using HL7formats, allowing the user to have access to analysis prior to review oforiginal X-rays. This standard process, ensures that individuals withsymptoms like cough over two weeks, diabetes and other immunocompromisedpatients get their screening for Tuberculosis done within seconds,referred for microbiological confirmation (NAAT) and if found positive,the treatment can start the same day. FIG. 4 shows the proposed workflowfor TB detection at district TB centers.

The deployment efforts will involve integrating with current X-raysystem through an API. The results are compatible with any radiologyviewer or can be hosted on a web-based the system viewer. The automatedreport is created with scores that correspond to a level of recognitionof the characteristics of the particular medical condition in the imagedata. Table 3 is an example of the automated report.

TABLE 3 An example of the automated report. Chest X-Ray abnormalitydetection and scoring X-Ray Findings Probability Remark Abnormal 0.94YES Blunted Costophrenic angle 0.24 NO Calcification 0.78 YESCardiomegaly 0.11 NO Cavity 0.85 YES Cervical Rib 0.42 NO Consolidation0.92 YES Hyper Inflation 0.44 NO Fibrosis 0.8 YES Prominence in Hilarregion 0.91 YES Opacity 0.95 YES Pleural Effusion 0.08 NO Scoliosis 0.1NO Tuberculosis screen 0.96 ADVISED

We claim:
 1. A method for developing a deep learning system to detectand localize medical abnormalities on chest X-ray imaging scans,comprising: selecting chest X-ray imaging scans and correspondingradiology reports and extracting the medical abnormalities via Naturallanguage processing (NLP) algorithms to generate extracted findings,wherein the extracted findings are used as labels for training a deeplearning algorithm; segmenting, via an anatomy segmenter, the selectedchest X-ray imaging scans to generate segmentation masks correspondingto chest cavity, lungs, diaphragm, mediastinum and ribs; outputting, viaa region of interest (ROI) generator, a plurality of ROIs that arerelevant for detecting a particular abnormality, wherein the ROIgenerator uses the chest X-ray imaging scan at full resolution and thecorresponding anatomy segmentation masks; and detecting theabnormalities, via abnormality detector, and outputting a low-resolutionprobability map per each ROI, a confidence score per each ROI, and aconfidence score for an entire chest X-ray scan by combining theconfidence scores per each ROI, wherein the abnormality detector is ahybrid classification plus segmentation network.
 2. The method of claim1, wherein the extracted findings comprise locations, severity, size,shape and texture.
 3. The method of claim 1, wherein the labelscomprises scan level labels, ROI level labels, and pixel level labels.4. The method of claim 1, wherein the NLP algorithms are rule-based. 5.The method of claim 1, wherein the deep learning algorithm comprisesconvolutional neural networks (CNNs).
 6. The method of claim 1, whereinthe anatomy segmenter uses a U-Net based neural network.
 7. The methodof claim 1, wherein the ROI generator is rule-based.
 8. The method ofclaim 1, wherein the confidence scores per each ROI are combined using apooling operator that is a convex approximation of LogSumExp (LSE)function.
 9. The method of claim 1, wherein the system uses a weightedsum of cross entropy loss at three levels comprising ROI levelpredictions, pixel level probability maps, and final pooled chest X-rayscan level prediction.
 10. The method of claim 1, wherein theabnormality detector is based on ResNeXT-50 with Squeeze-excitationmodules (SE-ResNeXT-50).
 11. The method of claim 1, wherein the medicalabnormalities comprise blunted CP angle, calcification, cardiomegaly,cavity, consolidation, fibrosis, hilar enlargement, opacity and pleuraleffusion.
 12. A system for automating detection and localization ofmedical abnormalities on chest X-ray imaging scans using a deep learningalgorithm carried out by a computer, wherein the deep learning algorithmis developed by the steps of: selecting chest X-ray imaging scans andcorresponding radiology reports and extracting the medical abnormalitiesvia Natural language processing (NLP) algorithms to generate extractedfindings, wherein the extracted findings are used as labels for traininga deep learning algorithm; segmenting, via an anatomy segmenter, theselected chest X-ray imaging scans to generate segmentation maskscorresponding to chest cavity, lungs, diaphragm, mediastinum and ribs;outputting, via a region of interest (ROI) generator, a plurality ofROIs that are relevant for detecting a particular abnormality, whereinthe ROI generator uses the chest X-ray imaging scan at full resolutionand the corresponding anatomy segmentation masks; and detecting theabnormalities, via abnormality detector, and outputting a low-resolutionprobability map per each ROI, a confidence score per each ROI, and aconfidence score for an entire chest X-ray scan by combining theconfidence scores per each ROI, wherein the abnormality detector is ahybrid classification plus segmentation network.
 13. The system of claim12, wherein the extracted findings comprise locations, severity, size,shape and texture.
 14. The system of claim 12, wherein the labelscomprises scan level labels, ROI level labels, and pixel level labels.15. The system of claim 12, wherein the NLP algorithms are rule-based.16. The system of claim 12, wherein the deep learning algorithmcomprises convolutional neural networks (CNNs).
 17. The system of claim12, wherein the anatomy segmenter uses a U-Net based neural network. 18.The system of claim 12, wherein the ROI generator is rule-based.
 19. Thesystem of claim 12, wherein the confidence scores per each ROI arecombined using a pooling operator that is a convex approximation ofLogSumExp (LSE) function.
 20. The system of claim 12, wherein the systemuses a weighted sum of cross entropy loss at three levels comprising ROIlevel predictions, pixel level probability maps, and final pooled chestX-ray scan level prediction.
 21. The system of claim 12, wherein theabnormality detector is based on ResNeXT-50 with Squeeze-excitationmodules (SE-ResNeXT-50).
 22. The system of claim 12, wherein the medicalabnormalities comprise blunted CP angle, calcification, cardiomegaly,cavity, consolidation, fibrosis, hilar enlargement, opacity and pleuraleffusion.